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WWC Review of the Report “Early College, Early Success: 
Early College High School Initiative Impact Study”^ 

The findings from this review do not ref iect the fuii body of research evidence on Early College High Schools. 


What is this study about? 

The study examined whether attending Early College 
High Schools improved high school and postsecond- 
ary outcomes for students. Early Colleges are high 
schools that partner with local colleges and universi- 
ties to offer students the opportunity to take courses 
towards earning an Associate’s degree or up to 2 years 
of college credit towards a Bachelor’s degree. The 
authors evaluated the impact of Early Colleges on high 
school outcomes, such as academic achievement, 
GPA, and graduation. In addition, the study examined 
the impact of Early Colleges on college enrollment, 
placement in developmental education, college GPA, 
and college degree attainment. The authors also 
reported on high school graduation, college enroll- 
ment, and college degree attainment for various sub- 
groups (females, racial/ethnic minorities, low income 
students, and first-generation college students).^ 

Using existing data collected and maintained by the 
schools, study authors identified ten Early Colleges 
in five states that (a) operated exclusively as Early 
College High Schools, (b) had graduates during the 
study years, and (c) employed and kept records of a 
lottery process to determine who would be offered 
admission to the program. A total of 2,458 students 
from 3 cohort years entered the identified lotteries. 
Students who were offered admission to Early Col- 
lege High Schools via the lottery formed the inter- 
vention group, while students not offered admission 
formed the comparison group. ^ The random assign- 
ment for this study was, therefore, not conducted by 
the study authors, but identified retrospectively from 
the participating schools’ lottery records. 


Features of Early College High Schools 


Early Colleges provide students with exposure to, 
and support In, college while they are In high school. 
Early Colleges partner with colleges and universities 
to offer all students an opportunity to earn an 
Associate’s degree or up to 2 years of college 
credits toward a Bachelor’s degree during high 
school at no or low cost to the students. 

Eight of the Early College High Schools In the study 
were typical 4-year high schools, and two Early 
Colleges had 5-year programs. Students had the 
opportunity to earn college credits or an Associate’s 
degree while completing their high school courses 
of study. 

Early Colleges are particularly Interested In 
Increasing college access and enrollment among 
students who are traditionally underrepresented In 
postsecondary Institutions, Including raclal/ethnic 
minorities, students from low socioeconomic status 
families, or those whose parents did not attend 
college. The Early College High Schools In this study 
served student populations that were, on average, 
49% raclal/ethnic minorities and 44% low Income. 
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What did the study find? 

Study authors reported, and the WWC confirmed, 
that 86% of Early College students graduated from 
high school by the end of the study period, com- 
pared to 81 % of comparison students, a statistically 
significant difference. The study also reported that 
students in Early Colleges had significantly better 
English/language arts achievement than students in 
comparison high schools; the WWC also confirmed 
this result. There were no significant differences 
between Early College High Schools and compari- 
son high schools on mathematics achievement or 
high school GPA. 

Postsecondary outcomes were collected through 
the end of the study period, with the cohorts fol- 
lowed for 2, 3, or 4 years after their expected gradu- 
ation date from high school. Study authors reported 
that postsecondary enrollment through the end of 
the study period was 80% for the Early College 
students and 71 % for the comparison students. 
Additionally, 22% of Early College students earned 
a postsecondary degree during the study period, 
compared to only 2% of comparison students. Early 
College students who enrolled in college were also 
less likely to be placed into developmental educa- 
tion than comparison students (18% vs. 22%). All of 
these differences were reported as being statistically 
significant, which the WWC confirmed. 

Among students who were enrolled in college for at 
least one term, there were no significant differences 
between Early College and comparison students in 
college GPA. 

The study reported high school graduation, college 
enrollment, and college degree attainment sepa- 
rately for males and females, racial/ethnic minorities 
and non-racial/ethnic minorities, and low income 
and non-low income students. The impact of Early 
Colleges on high school graduation and college 
enrollment was similar for males and females, for 
racial/ethnic minorities and non-racial/ethnic 
minorities, and for low income and non-low income 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

This study is a well-executed randomized 
controlled trial. The Impact estimates for high 
school achievement in mathematics and English/ 
language arts, high school GPA, high school 
graduation, college enrollment, and college degree 
attainment meet WWC evidence standards without 
reservations. 

The impact estimates for college GPA, placement in 
developmental education in college, and persistence 
in college for subsamples of students were 
evaluated by the WWC as quasi-experimental 
comparisons because they were derived from 
non-random subsets from the original randomized 
sample; therefore, they were required to 
demonstrate baseline equivalence. Program impacts 
on college GPA and placement in developmental 
education were estimated only for students who 
had enrolled in college. These outcomes meet 
WWC evidence standards with reservations. 

Program impacts on persistence in college were 
estimated only for students who enrolled in college; 
this outcome does not meet WWC evidence 
standards because baseline equivalence could 
not be established. 


students. However, the impact of Early Colleges on 
college degree attainment was significantly stronger 
for female students, racial/ethnic minorities, and low 
income students. These findings were confirmed by 
the WWC (for more information, see Appendix D). 
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Appendix A: Study details 

American Institutes for Research & SRI. (2013). Early college, early success: Early College High School 
Initiative Impact study. Washington, DC: American Institutes for Research. Retrieved from 
http://www.air.org 


Setting 

The ten Early Colleges in the study were located in five states throughout the country: five in 


large urban areas, two in mid-sized cities, and three in smail towns. Eight of the ten Early 
Colleges were located on college campuses. Seven had a 2-year public college partner, two 
had a 4-year public college partner, and one had both. 

Study sample 

The sample consisted of general education high school students. About half (52%) of the Early 
College group was female, vs. 55% of the comparison group. Racial/ethnic minority students 
comprised 52% and 54% of the intervention and comparison groups, respectively. In addition, 

31 % of the intervention group were first-generation college students, vs. 34% of the comparison 
group. Low income students comprised 47% of the intervention group, vs. 42% of the compari- 
son group. None of these differences in group composition was statistically significant. 

Intervention 

group 

Six Early Colleges were district-run schools, and the remaining four were charter schools. 
Most of the schools had a particular academic focus in addition to being Early Colleges: five 
had a STEM focus, and two had a teacher preparation focus. The Early Colleges offered a 
number of supports, with all ten Early Colleges providing tutoring and college preparatory 
information. In addition, some of the Early Colleges offered regular advisory meetings with 
counselors; summer, evening, and weekend classes; extended school days; and/or block 
scheduling. In terms of the college coursework, seven Early Colleges had course sequences 
that allowed students to earn at least 2 years of college credit, two Early Colleges allowed 
students to earn up to 1 year of college credit, and one Early College had a course sequence 
that allowed students to earn at least some college credit. 

Comparison 

group 

The comparison students in the study attended 272 different high schools. The majority of the 
students who did not attend Early Colleges enrolled in larger high schools with larger racial/ 
ethnic minority and low income student populations. Those schools provided fewer academic 
supports (e.g., tutoring) and a less direct focus on college readiness for all students. AP 
courses seemed to be more prevalent than dual enrollment as a strategy for students to earn 
college credit. 
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Outcomes and 
measurement 


Support for 
implementation 

Reason for 
review 


The study reported both high school and postsecondary outcomes. The eligible high school 
outcomes are: standardized English/language arts achievement, standardized mathemat- 
ics achievement, high school GPA, and high school graduation. The eligible postsecondary 
outcomes are: any college enrollment, any postsecondary degree, placed In developmental 
education in college, and college GPA. The data for the high school graduation outcomes 
were obtained from high school records maintained by a variety of local sources. High school 
standardized achievement tests in English/language arts and mathematics were obtained from 
high school administrative records. High school GPAs were obtained from a student survey. 
The data for college enrollment and degree attainment outcomes were obtained from the 
National Student Clearinghouse (NSC). The NSC data provide conservative estimates of col- 
lege enrollment and degree attainment. If a student did not have a record in the NSC, then that 
student was coded as not being in college. However, students may have been missing from 
the NSC because (a) they attended a college that did not provide data to NSC, (b) they did not 
allow NSC to share their individual record data, or (c) their name or birthday did not match the 
information in the NSC. Study authors stated that they had no reason to expect NSC data to 
be differentially missing for Intervention and comparison students. The data for college GPA 
and developmental placement In college were obtained from a student survey. For a more 
detailed description of these outcome measures, see Appendix B. 

All but one of the Early Colleges had college Instructors, rather than qualified high school 
instructors, teaching college courses. 

This study was identified for review by the WWC by receiving significant media attention. 
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Appendix B: Outcome measures for each domain 


High school degree attainment 

High school graduation 

To measure high school graduation, study authors used high school administrative data. The source of the data 
differed by research site, in addition, study authors used data from muitipie sources for some sifes. 

High school achievement in English/language arts 

Achievement in English/language arts 

To measure achievement in Engiish/language arts in high school, study authors used high school administrative 
data. The source of the data differed by research site, in addition, study authors used data from multiple sources 
for some sites. Scores were standardized test scores, typically from tenth grade, that were standardized using 
the state mean and standard deviation for each year and grade level. 

High school achievement in mathematics 

Achievement in mathematics 

To measure achievement in mathematics in high school, study authors used high school administrative data. The 
source of the data differed by research site. In addition, study authors used data from multiple sources for some 
sites. Scores were standardized test scores, typically from tenth grade, that were standardized using the state 
mean and standard deviation for each year and grade level. 

High school GPA 

High school GPA 

To measure high school GPA, study authors used information from a survey conducted with a sample of stu- 
dents who participated in the initial randomization. GPA was indexed using a 0-4 variable indicating students’ 
self-reported high school grade point average. 

College enrollment 

College enrollment 

To measure enrollment, study authors used the National Student Clearinghouse (NSC) to determine whether 
or not students enrolled in college. The follow up period varied, with the three cohorts followed up to 2, 3, or 4 
years post-high school. Study authors linked administrative data from the participating schools, districts, and 
states to the NSC postsecondary data. For study students who attended colleges that contributed to the NSC 
and allowed their records to be made available, data on college enrollments were obtained. Students not located 
in the NSC were assumed to not have enrolled in college. 

College degree attainment 

College degree attainment 

To measure degree attainment, study authors used the NSC to determine whether or not students earned 
any postsecondary credential, including certificates. Associate’s degrees, or Bachelor’s degrees. The follow 
up period varied, with the three cohorts followed up to 2, 3, or 4 years post-high school. Study authors linked 
administrative data from the participating schools, districts, and states to the NSC postsecondary data. For 
study students who attended colleges that contributed to the NSC and allowed their records to be made 
available, data on degree attainment were obtained. Students not located in the NSC were assumed to not have 
enrolled in college or attained a degree. 

Developmental education 

Placement in developmental education 

Among students who enrolled in college, a dichotomous indicator of whether a student enrolled in any develop- 
mental education in college was obtained from the student survey. 

College GPA 

College GPA 

Among students enrolled in college for at least one term, the student survey asked about college GPA. The 
variable used was a 0-4 indicator of college GPA after high school. 


Table Notes: Thirteen other outcomes were examined in this study, but are not inciuded in this report. Coiiege persistence was an eiigibie outcome under the protocoi and was 
reported on a subsampie of students who were initiaiiy randomized; the WWC considers these to be quasi-experimentai comparisons, for which baseiine equivaience must be 
estabiished. Group equivaience was not estabiished for the coiiege persistence outcome. Therefore, persistence in coiiege does not meet standards and is not inciuded in this 
report, in addition, the foiiowing five outcomes were reported but are not eiigibie: enroiiment in a 2-year coiiege, enroiiment in a 4-year coiiege, attainment of certificate, attain- 
ment of Associate’s degree, and attainment of Bacheior’s degree. The review team determined that these five outcomes were not eiigibie because the authors dichotomized essen- 
tiaiiy ordinai variabies in ways that combined positive outcomes with negative outcomes (e.g., for the Associate’s degree attainment outcome, no degree attainment was combined 
with certificate and Bacheior’s degree attainment for the comparison), in addition, the study aiso reported on student perceptions of academic difficuify in coiiege, academic rigor 
in high schooi, instrucfionai rigor in high schooi, and supportive resources in high schooi; these four outcomes are not eiigibie under the Postsecondary Education or singie study 
review protocois. Finaiiy, three outcomes were reported that the review team determined were overaiigned with either the intervention or comparison conditions. These inciuded: 
enroiiment in coiiege whiie in high schooi (the intervention is enroiiment in coiiege whiie in high schooi), coiiege credit accumuiation whiie in high schooi (overaiigned with the 
intervention schoois), and AP course taking in high schooi (highiy aiigned with comparison group participation). 
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Appendix C: Study findings for each domain 




Mean 

(standard deviation) 


WWC calculations 


Domain and Study 

outcome measure sample 

Sample 

size 

Intervention Comparison 
group group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

High school degree attainment 

High school graduation Al 1 

students 

2,458 

students 

86% 

81% 

5% 

0.24 

+9 

<0.05 

Domain average for high school degree attainment 




0.24 

+9 

Statistically 

significant 

High school achievement in English/language arts 

Achievement in English/ All 

language arts students 

2,141 

students 

0.37 

0.23 

0.14 

0.15 

+6 

<0.05 

Domain average for high school achievement in English/language arts 



0.15 

+6 

Statistically 

significant 

High school achievement in mathematics 

Achievement in All 

mathematics students 

1,628 

students 

0,28 

0.23 

0.05 

0.06 

+2 

0.19 

Domain average for high school achievement in mathematics 



0.06 

+2 

Not 

Statistically 

significant 

High school GPA 

High school GPA All 

students 

1,273 

students 

2,98 

2,98 

0 

0.00 

0 

0,94 

Domain average for high school GPA 





0.00 

0 

Not 

Statistically 

significant 

College enrollment 

College enrollment All 

students 

2,458 

students 

80% 

71% 

9% 

0.30 

+12 

< 0.0001 

Domain average for college enrollment 





0.30 

+12 

Statistically 

significant 

College degree attainment 

College degree attainment All 

students 

2,458 

students 

22% 

2% 

20% 

1.63 

+45 

< 0.0001 

Domain average for college degree attainment 




1.63 

+45 

Statistically 

significant 

Developmental education 

Placement in developmental Students 
education enrolled In 

college 

1,002 

students 

18% 

22% 

4% 

0.14 

+6 

0.024 

Domain average for developmental education 




0.14 

+6 

Statistically 

significant 
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College GPA 

College GPA Students 

enrolled for 
at least 1 
college term 

455 

students 

3.07 

3.09 

0.02 

-0.02 

-1 

0.944 

Domain average for college GPA 





-0.102 

-1 

Not 

Statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for aii students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the 
change in an average student’s percentiie rank that can be expected if the student is given the intervention. The statisticai significance of the study's domain average was deter- 
mined by the WWC; the study is characterized as having a statisticaiiy significant positive effect because the effect reported is positive and statisticaiiy significant. 

Study Notes: No corrections for ciustering or muitipie comparisons were needed. The p-vaiues presented here were reported in the originai study. The impact estimates reported 
in the tabie are derived from modeis that adjusted for ciustering of students within iotteries and inciuded the foiiowing covariates: femaie, raciai/ethnic minority, first-generation, 
iow income, prior achievement in Engiish/ianguage arts, and prior achievement in mathematics, information on baseiine equivaience for the deveiopmentai education and coiiege 
GPA outcomes was obtained from the authors. Aii other information reported in the tabie was taken from the study report. 
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Appendix D: Supplemental findings by domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

jo-value 

High school degree attainment 

High school graduation 

Female 

1,264 

students 

85% 

83% 

2% 

0,08 

- h 3 

0,15 

High school graduation 

Male 

1,194 

students 

87% 

78% 

9% 

0,40 

-Hi 5 

<0,001 

High school degree attainment 

High school graduation 

Racial/ethnic 

minorities 

965 

students 

87% 

82% 

6% 

0,26 

-HlO 

<0,001 

High school graduation 

Non-racial/ 
ethnic minorities 

855 

students 

89% 

83% 

7% 

0,33 

-Hi 3 

<0,001 

High school degree attainment 

High school graduation 

Low income 
students 

1,263 

students 

83% 

74% 

9% 

0,32 

-Hi 3 

<0,001 

High school graduation 

Non-iow income 
students 

1,125 

students 

89% 

87% 

2% 

0,12 

- h 5 

0,051 

College enrollment 

College enrollment 

Female 

1,264 

students 

81% 

75% 

6% 

0,22 

- h 9 

<0,001 

College enrollment 

Male 

1,194 

students 

78% 

66% 

13% 

0,39 

-Hi 5 

<0,001 

College enrollment 

College enrollment 

Racial/ethnic 

minorities 

965 

students 

80% 

72% 

8% 

0,27 

-Hll 

<0,001 

College enrollment 

Non-racial/ 
ethnic minorities 

855 

students 

83% 

73% 

10% 

0,37 

-Hi 4 

<0,001 

College enrollment 

College enrollment 

Low income 
students 

1,263 

students 

75% 

64% 

11% 

0,31 

-Hi 2 

<0,001 

College enrollment 

Non-low income 
students 

1,125 

students 

85% 

76% 

9% 

0,35 

-Hi 4 

<0,001 

College degree attainment 

College degree 
attainment 

Female 

1,264 

students 

23% 

1% 

22% 

1,93 

- h 47 

<0,001 

College degree 
attainment 

Male 

1,194 

students 

22% 

3% 

19% 

1,35 

- h 41 

<0,001 
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College degree attainment 


College degree 
attainment 

Raciai/ethnic 

minorities 

965 

students 

65% 

1% 

64% 

3.23 

-f50 

<0.001 

College degree 
attainment 

Non-racial/ 
ethnic minorities 

855 

students 

23% 

3% 

20% 

1.41 

-f42 

<0.001 

College degree attainment 

College degree 
attainment 

Low income 
students 

1,263 

students 

20% 

1% 

19% 

2.07 

-h48 

<0.001 

College degree 
attainment 

Non-iow income 
students 

1,125 

students 

25% 

3% 

21% 

1.35 

-f41 

<0.001 


Table Notes: For mean difference, effecf size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for aii students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the 
change in an average student’s percentiie rank that can be expected if the student is given the intervention. 

Study Notes: No corrections for ciustering or muitipie comparisons were needed. The p-vaiues presented here were computed by the WWC using the information reported in 
Exhibit E.5. The impact estimates reported in the tabie are derived from modeis that adjusted for ciustering of students within iotteries and covariates. The modeis were run using 
ten muitipie imputation datasets; WWC attrition standards were met using unimputed sampie sizes provided by the authors. The sampie sizes reported in the tabie are the means 
across the ten imputed datasets, rounded to the nearest whoie number, and were provided by the authors. For high schooi graduation and coiiege enroiiment, the differences in 
impact estimates were not significantiy different for maies and femaies, raciai/ethnic minorities and non-raciai/ethnic minorities, or iow income and non-iow income students. The 
differences in impact estimates for coiiege degree attainment were statisticaiiy significantiy different for maies and femaies, raciai/ethnic minorities and non-raciai/ethnic minori- 
ties, and iow income and non-iow income students. The study aiso reported on the impact of Eariy Coiieges for first-generation coiiege students vs. non-first-generation coiige 
students. This subgroup anaiysis did not meet WWC standards; attrition for the subgroups was high, and sufficient information to estabiish baseiine equivaience was not avaiiabie. 
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Endnotes 

■' Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
author[s]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Postsecondary Education topic area review protocol, version 2.0. A quick review of 
this study was released on August 2, 2013, and this report is the follow up review that replaces that initial assessment. 

^ Thirteen other outcomes were examined in this study, but are not included in this report. College persistence was an eligible out- 
come under the protocol and was reported on a subsample of students who were initially randomized; the WWC considers these to be 
quasi-experimental comparisons, for which baseline equivalence must be established. Group equivalence was not established for the 
college persistence outcome. Therefore, persistence in college does not meet standards and is not included in this report. In addition, 
the following five outcomes were reported but are not eligible: enrollment in a 2-year college, enrollment in a 4-year college, attain- 
ment of certificate, attainment of Associate’s degree, and attainment of Bachelor’s degree. The review team determined that these five 
outcomes were not eligible because the authors dichotomized essentially ordinal variables in ways that combined positive outcomes 
with negative outcomes (e.g., for the Associate’s degree attainment outcome, no degree attainment was combined with certificate and 
Bachelor’s degree attainment for the comparison). In addition, the study also reported on student perceptions of academic difficulty in 
college, academic rigor in high school, instructional rigor in high school, and supportive resources in high school; these four outcomes 
are not eligible under the Postsecondary Education or single study review protocols. Finally, three outcomes were reported that the 
review team determined were overaligned with either the intervention or comparison conditions. These included: enrollment in college 
while in high school (the intervention is enrollment in college while in high school), college credit accumulation while in high school 
(overaligned with the intervention schools), and AP course taking in high school (highly aligned with comparison group participation). 

® This single study review reports results of an intent-to-treat analysis. Students who were offered admission into an Early College 
High School via winning a lottery formed the intervention group, regardless of whether they attended an Early College. Students who 
were not offered admission via a lottery formed the comparison group, regardless of where they attended high school. Across the ten 
study sites, 22% of the students offered admission to an Early College High School did not enroll; these students were included in the 
intervention condition. Two percent of the students who were not offered admission to an Early College High School actually enrolled 
in one; these students were included In the comparison condition. WWC standards require that reviews be based only on the Intent- 
to-treat analyses because they preserve the benefits of random assignment and provide internally valid estimates of program impacts. 
The study authors report the results of the compiler (or treatment-on-the-treated) analyses in Appendix G. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, March). WWC 
review of the report: Early college, early success: Early College High School Initiative impact study. Retrieved 
from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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